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On  convergence  of  Weak  Greedy  Algorithms1 


E.D.  Livshitz  and  V.N.Temlyakov 

Moscow  State  University,  Moscow,  Russsia/ / 
University  of  South  Carolina,  Columbia,  SC,  USA 


1.  Introduction 

This  paper  is  devoted  to  investigation  of  Weak  Greedy  Algorithms  (WGA)  intro¬ 
duced  in  [T].  We  remind  some  notations  and  definitions  from  the  theory  of  greedy 
algorithms.  Let  H  be  a  real  Hilbert  space  with  an  inner  product  (-,  •)  and  the  norm 
||x||  :=  (a:,^)1/2.  We  say  a  set  V  of  functions  (elements)  from  H  is  a  dictionary  if 
each  geD  has  norm  one  (||g||  =  1)  and  span©  =  H.  We  give  now  the  definition  of 
WGA  (see  [T]).  Let  a  sequence  r  =  {tk}%T11  0  <  tk  <  1,  be  given. 

Weak  Greedy  Algorithm.  We  define  /q  :=  /.  Then  for  each  m  >  1,  we  induc¬ 
tively  define: 


!)■ 

ipTm  E  is  any  satisfying 

1  (frn—l  >  Trn)  1 

>  tm  SUp  |(/£_i,0)| 

g£V 

2). 

fT  . _  £T 

J  m  *  J m— 

1  {f m—  D  TmfTmi 

3). 

777, 

GTm(f,V): 

i= 1 


The  following  theorem  has  been  proven  in  [T]. 

Theorem  A.  Assume 

a,,  ££  =  »• 

k= 1 

Then  for  any  dictionary  V  and  any  f  e  H  we  have 

lim  ||/  —  G1m(f,/D)\\  =  0. 

m—¥  oo 

In  Section  2  of  this  paper  we  prove  the  following  theorem. 


1This  research  was  supported  by  the  National  Science  Foundation  Grant  DMS  9970326  and 
by  ONR  Grant  N00014-91-J1343 
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Theorem  1.  In  the  class  of  monotone  sequences  r  =  1  >  ti  >  t-2  >  ■  ■  ■  > 

0,  the  condition  (1.1)  is  necessary  and  sufficient  for  convergence  of  Weak  Greedy 
Algorithm  for  each  f  and  all  Hilbert  spaces  H  and  dictionaries  V. 

In  Section  3  we  consider  another  particular  case  of  sequences  r.  Let  A f  := 
{nk}ff1,  ni  <  n2  <  . .  • ,  be  a  given  subsequence  of  natural  numbers  and  0  <  t  <  1. 
We  define 

r(Af,t)  :=  {tn  :  tnk  =  t  and  tn  =  0,  nk  <  n  <  nk+1,  k  =  1,2...}. 

It  is  convenient  for  us  to  impose  some  regularity  restrictions  on  A f.  We  consider 
the  class  A4  of  sequences 

M  :=  {{nk}fLi  ■  nk+1  -  nk  >  nk  -  nfc_i;  nk+1nk^  <  n2k,  k  =  2,...}. 


We  prove  in  Section  3  the  following  theorem. 

Theorem  2.  In  the  class  of  sequences  r(Af,t),  A f  E  Ai,  the  condition 


(I) 


OO 


E 


(nk+i  ~  nk)1/2 
nk 


=  oo 


is  necessary  and  sufficient  for  convergence  of  Weak  Greedy  Algorithm  for  each  f 
and  all  Hilbert  spaces  H  and  dictionaries  V. 


2.  Proof  of  Theorem  1 

The  sufficiency  of  condition  (1.1)  for  convergence  follows  from  Theorem  A.  We 
prove  here  the  necessity  part  in  Theorem  1.  Let  H  be  a  Hilbert  space  with  an 
orthonormal  basis  {ej}°?=1.  For  two  elements  e,;,  ej,  i  /  j,  and  for  a  positive 
number  t  <  1/3  we  define  the  procedure  which  we  call  ”  equalizer”  and  denote 
,  ej,t). 

Equalizer  E(ei,  ej,  t).  Denote  /o  :=  e*  and  g\  :=  aqej  —  (1  —  «f)1/2ej  with  a\  :=  t. 
Then  ||#i||  =  1  and  (/0,£i)  =  t.  We  define  the  sequences  92 ,  — , 

ct2,  •  •  • ,  ccn  inductively: 

fn  ■  fn  —  1  (fn—  1)  9n)9ni  9n+ 1  •  (1  ^ 

with  o:n+i  satisfying 

(fm  9n+ 1)  Tl'  1,2,.... 

Let  fn  =  anei  +  bnej  and  N  :=  Nt  be  the  number  such  that 

a jv— i  —  6 jv— l  E  \/2t,  a at  —  <  \Z2L 

Then  we  modify  the  N- th  step  as  follows.  We  take  Qn  ■=  2 _1/2(ej  —  ej)  and 

/a  =  /jV-l  “  (fN-l,9N)dN- 
It  is  clear  that  than  a  at  =  6/v  and 


t  <  (fN-i,  9n )  <  2 1. 
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We  list  here  the  following  simple  relations 

^n+l  Q"n  ^^n+1?  ^n+1  H”  ^(1  C^n_j_i)  ^  5  77/  <C  iV"  1? 


(2.1)  nn_i_i  6n_i_i  &n  bn  t((an_|_i  H-  (1  o^n_|_i)  ^  ti  <C.  N  1, 

||/n+l||2  =  ||/n||2-t2,  n<N-l. 

Relation  (2.1)  and  the  inequality  1  <  x  +  (1  —  a:2)1/2  <  21/2,  0  <  x  <  1,  imply  that 

(2.2)  N  <  1/t 
and 

ll/ivll2  >  ||/iv-i||2  -  4t2  >  ||/||2 
This  gives  for  t  <  1/3  that 

ll/ivll2  >  ll/ll2  -  2t. 

It  is  clear  that  E(e i,  ej,t)  is  a  WGA  with  regard  to  the  dictionary  e*,  gi,  g2,  ■  ■  ■ ,  gN 
with  the  ” weakness”  parameter  t. 

Let  1/3  >  ti  >  t2  >  •  •  •  >  0  be  such  that 


(2.3) 


with  e  >  0  to  be  chosen  later.  Then 


^  '  ^2 s  ^  2e. 

s=0 


We  define  WGA  and  a  dictionary  V  as  follows.  We  begin  with  /  :=  e\  and  apply 

E(e i,  e2,  t).  After  Ntl  >  1  steps  we  get  g® , . . .  ,g%  and 

*1 


/ 1  =  G(ei  +  e2) 


with  the  properties 

II/1!!2  >  ll/ll2  -2ti;  (Cl)2<i/2. 

We  use  now  _E(ei,  e3, £2)  and  E(e 2,  e4,  £2)-  After  2W2  >  2  steps  we  obtain  g\, . . . ,  glN 

*2 

and 

f2  =  c2(e  1  H - +  64) 

with  the  properties 

||/2||2  >  II/1!!2  -  2t2;  (c2)2<2-2. 


After  s  iterations  we  get 


/s  —  cs(e  !  +  •••  +  e2s) 
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and  apply  E(ei,ei+ 2s,t2s),  i  =  1,2,...,  2s.  We  make  2  sNt23  >  2s  steps  and  get 

9\->  -  ■  ■  j  92s Nt  and 
L2S 

fs+1  =  cs+i(ei  +  •  •  •  +  e2s+i) 


with  the  properties 

OO 

Iir+1||2  >  ll/ll2 -2ti -2t2 - 2 t2s  >  l-2^t2fl  >  l-4e. 

S  =  1 

(cs+1)2  <  2—\ 

Choosing  e  =  ^  we  see  that  \\fs\\  >  1/2  for  all  s. 

Thus  we  get  that  the  WGA  with  r  satisfying  (2.3)  does  not  converge  for  /  =  e i 
with  regard  to  the  dictionary 

®=Ue‘u  U 

fceN  s>0;i<i<2a  Nt23 


We  will  show  now  how  the  general  case 


can  be  reduced  to  the  case  (2.3).  We  find  n  such  that 

OO 

^t2s  <  e, 

s=n 


take  /  =  eiH - be2«  and  pick  at  the  first  2n  —  1  steps  ei, ,  e2«_i  as  approximating 

elements  from  the  dictionary.  Then  we  use  the  described  above  procedure  with 
/  =  e2»  instead  of  e\  with  the  natural  change  in  indices. 


3.  Proof  of  Theorem  2 
We  consider  here  the  case  of  r  =  {tn}^=1  of  the  form 

(3.1)  tnk=t  and  tn  =  0,  nk  <  n  <  nk+ 1,  A:  =  1,2..., 

for  a  given  subsequence  n\  <  n2  <  . . . .  Theorem  A  implies  that  the  Weak  Greedy 
Algorithm  with  the  above  r  converges  if 


OO 

E 1/nfc  =  °°- 

k= 1 


Theorem  2  shows  that  the  above  condition  can  be  replaced  by  a  weaker  one.  We 
begin  with  the  proof  of  the  sufficiency  part  of  Theorem  2. 
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Lemma  3.1.  In  the  class  M.  of  sequences 
M  :=  {{nk}™=1  :  nk+1  -  nk  >  nk  -  nk-i;  nk+1nk^<n\,  k  =  2,...} 

the  following  two  conditions  are  equivalent 


(I) 


OO 


E 


{nk+ i  ~  nk)x!2 
nk 


=  oo; 


(II) 


nk 

V{aj}  £  I2  liminfanfe  N  aj  =  0. 

k— >00  ^ ' 

3  = 1 


Remark  3.1.  We  point  out  here  that  in  the  proof  of  (I)  (II)  in  Lemma  3.1  we 
use  only  the  property  of  boundedness  of  nk+i/nk: 

(B)  3C  :  Wke  N,  <  C. 

nk 

Thus  in  the  sufficiency  part  of  Theorem  2  the  assumption  {nk}(jfL1  £  A4  can  be 
replaced  by  a  weaker  assumption  (B).  We  note  also  that  in  the  proof  of  (II)  (I) 
in  Lemma  3.1  we  use  only  the  property  of  convexity: 

(C)  nk  ^  nk  nk— i- 

Proof  of  Lemma  3.1.  Let  us  prove  first  that  (I)  implies  (II).  We  will  prove  the 
following  a  little  stronger  statement  than  (II) 

OO  Tlk 

(3.2)  ^2(nk+ 1  -  nk)1/2nk  1\ank\  ^  \aj\  <  00. 

k= 1  j=l 

It  is  known  (see  [Z],Ch.l,S.9)  that  {aj}JL1  £  I2  implies  that 

1  n 

(3.3)  {K}^Li  £  1-2  with  bn  :=  -  'V'  \aj\. 

n 

3= 1 

We  observe  first  that 

OO 

(3.4)  ^2(nk+ 1  -  nk)b2nk  <  00. 

k=  1 

Indeed,  for  any  m  >  n  we  have 


mbm  >  nbn 

and  for  nk  <  m  <  nk+ 1  we  have  for  {nk}(^=1  £  A A 

b 


/  ^k+l  ^  S'*], 

rife  _  Om.  ^  V? 


nk 
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with  a  constant  C  independent  of  k  and  to.  Therefore 

n-k+ 1  —  1 

(3-5)  [nk+ 1  -  nk)b2nk  <  C2  b m- 

m=rik 


Combining  (3.3)  and  (3.5)  we  get  (3.4). 

We  return  to  (3.2) 

OO  rife  OO 

J2{nk+i  -nfc)1/2nfc1|anfe|J]|aJ|  =  ^{nk+1  -  nk)1/2\ank \bn.k  < 

k=  1  j  =  l  fc=l 

OO  OO 

EanJ1/2E(nfc+l  -Wfc)6^J1/2  <  OO. 

fc=l  fc=l 

Let  us  prove  now  that  (II)  implies  (I).  Assume  the  contrary  that  for  {nk}<^=1  G  A4 
we  have 

OO 

(3.6)  y^(nfc+i  -  nfc)1/2nfc  1  <  oo. 

k=i 

We  will  construct  a  sequence  G  h  such  that 

(3.7)  liminfa„,fe  N  (ij  >  0. 

i=i 


Define 


for 


:=  (rifc+i  -  nk)1/4nk  1/2;  aj  :=  (nfc+i  -  rafc)  1/4nfc  1/2, 


j  G  (nfc,nfe+i),  k  =  1,2,.... 

Then  (3.6)  implies  that  G  h-  We  have  from  the  definition  of  that 


^  Oj  >  (nfc+i  -  nfc)3/4 

iG[nfe,nfe+i) 


n. 


1/2 


Next,  we  obtain  from  here 

^■fe+i  fc  fc 

ai -  5Z  -  ni)3/4n71/ 2  > 

j=l  2=1  jG[ni,ni+i)  2=1 


Ai  Ai 

y^(nfc+i  -  nfc)~1/4(?p+i  -  ni)n^1/2  >  {nk+1  -  nfc)~1/4  ^(rqj2  -  nj/2)  = 
2=1  2=1 

(rcjfc+i  -  nfc)-1/4(n4/+2i  -  n\/2). 

This  estimate  and  the  definition  of  aUk  implies  (3.7).  Lemma  3.1  is  proved  now. 


7 


Lemma  3.2.  Assume  that  a  sequence  {rik}<^=1  G  M.  satisfies  (I).  Let  0  <  t  <  1 
and  t  =  {tn}^=1  satisfies  (3.1).  Then  {fnk-i}kLi  converges. 

This  lemma  combined  with  the  following  simple  modification  of  Lemma  2.1  from 
[T]  give  the  sufficient  part  of  the  conclusion  of  Theorem  2. 

Lemma  3.3.  Assume  that  for  some  {nk}(fi=1 

OO 

ZX  =  o°- 

k=  1 

Then  if  {ffk-i}kLi  conveges  it  converges  to  zero. 

Proof  of  Lemma  3.2.  This  proof  is  similar  to  the  corresponding  arguments  from 
[T].  We  present  it  here  for  selfcompleteness  of  this  paper.  It  is  easy  to  derive  from 
the  definition  of  WGA  the  following  two  relations 

m 

(3.8) 

3  = 1 

m 

(3-9)  ll/J|2  =  ll/ll2 

3  =  1 

Denote  aj  :=  |(/J_l5^J)|.  We  get  from  (3.9)  that 

OO 

Ed  <  ii/ii2- 

3  = 1 

We  take  any  two  indecies  n  <  m  and  consider 

ii/:  -  02  =  ii/:ii2  -  no2  -  2 </:  -  rm,  rj. 

Denote 

:=  l(/:  - 

Using  (3.8)  and  the  definition  of  the  WGA  we  get  for  all  n  <  m  that 

m  ra+1 

(3.io)  E  IU7-i.^}M</m.^}l<r^E<v 

j=n+ 1  m+1  j  =  l 

Specifying  n  =  n;  —  1  and  m  =  —  1  we  get  from  (3.10)  that  for  any  l  <  k 

nk 

(3-11)  @ni-l,nk-l  —  t  ank  ^  ]aj- 

3  =  1 

The  relation  (3.11)  and  Lemma  3.1  imply  that 

lim  maxtU  i  ■,  =  0. 
l<k  1 


It  remains  to  use  the  following  simple  lemma  (see  [T]). 
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Lemma  3.4.  Let  in  a  Banach  space  X  a  sequence  {xn}^=1  be  given.  Assume  that 
for  any  k,  l  we  have 

II xk  -  xi ||2  =  yk  -yi  +  9k, i, 

with  {yn}n Li  is  a  convergent  sequence  of  real  numbers  and  6k,i  satisfying  the  prop¬ 
erty 

lim  max  6k  i  =  0. 

k<1  ’ 


Then  {x„}“=1  converges. 

We  proceed  now  to  the  necessity  part  of  Theorem  2.  We  will  need  the  following 
simple  properties  of  sequences  from  A 4.  Denote 

An k  ■ —  ^fc+i  ii'k  ■ 

Then  by  monotonicity  of  {An;-}  we  have 


2s  —  1 

(3.12)  n2=  —  n2s- 1  =  A nk  <  2s~1An2« 

k=2s~1 


and 


2s-1-! 

n2s  —  n2s~ i  >  A nk  =  n2«-i  —  «i . 

k=  1 


(3.13)  n2s  —  n2B-i  >  (n2«  —n\)/2. 

Combining  (3.12)  and  (3.13)  we  get 

An2s  >  2 ~s(n2s  —  m) 


and 


(3.14)  (A n2s)  <  2s(An2s)1/2(n2s  —  m) 

Next,  {(Arik)1/2 /nk}  is  a  monotone  sequence: 

(A  nk)1/2/nk  =  nk1/2(Ank/nk)1/2,  nk  t,  A nk/nk  f  . 
Thus  the  following  two  conditions  are  equivalent 


y-  (Anfc)1/2 

A—/  n  .7. 


<  CXD, 


os  (An2s)1/2 


(3.15) 


<  X. 
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It  is  clear  that  (3.14)  and  (3.15)  imply 


y^(An2«)  1//2  <  oo. 

s= o 

The  construction  of  the  corresponding  counterexample  is  similar  to  that  from  Sec¬ 
tion  2.  We  assume  that 

OO 

y^(An2«)~1/2  <  e 

s= 0 

with  small  enough  e,  say,  e  =  3/16.  Define  a  new  sequence  r'  :=  {t'n}^=1  with 

C  =  (An  2°)~1/2,  ne[  2S,2S+1),  s  =  0,1,.... 

Then  t'n  0  and  the  WGA  from  Section  2  with  r'  and  /  :=  e\  reduces  the  square 
of  the  norm  of  /  by  at  most  2e.  We  modify  now  the  above  WGA.  At  each  step  ny. 
we  replace  WGA  by  Pure  Greedy  Algorithm  (PGA)  what  means  that  at  each  step 
rtk  we  throw  away  a  term  q Cj  with,  say,  the  biggest  j.  Let  us  estimate  how  much 
do  we  reduce  the  square  of  the  norm  of  /  in  this  way.  After  s  iterations  we  have 

fs  =  cs(e i  +  •  •  •  +  ejva),  Ns  <  2s,  (cs)2  <  2  s . 


Relation  (2.2)  implies,  that  working  on  ( s  +  l)-st  iteration  we  will  make  at  most 
Ns/t'2s  steps  of  the  algorithm.  It  is  easy  to  see  that  after  s  iterations  we  have  made 
at  least  2s  steps.  Therefore  we  will  use  PGA  at  most 


TV,  2s 

t2sAri2‘  —  (A n2s)1/2 


times  during  (s  +  l)-st  iteration.  Thus  the  norm 


will  be  reduced  by  at  most 


1 

(A  n^)1/2 


(l  +  2~s) 


what  gives  the  total  reduction  due  to  PGA  steps  at  most  2e.  This  reduction  com¬ 
bined  with  the  reduction  of  WGA  at  other  steps  sums  up  to  at  most  4e.  Choosing 
e  small  enough  (say  e  =  3/16)  we  get  divergence  of  the  defined  above  WGA. 

Remark  3.2.  In  the  proof  of  the  necessity  part  of  Theorem  2  we  have  used  the 
convexity  property  (C)  and  also  the  monotonicity  of  {(Ank)1^2 /nk}  what  is  weaker 
than  the  assumption  {nk}<£=1  6  A4. 
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